feat(telemetry): expose OTel metrics via a Prometheus scrape endpoint by cdoern · Pull Request #6034 · ogx-ai/ogx

cdoern · 2026-06-04T19:56:29Z

What

Adds an optional metrics scrape endpoint that exposes the existing OTel metrics in Prometheus exposition format, alongside the current OTLP push path. This unblocks scrape-based monitoring systems that need a scrape endpoint rather than OTLP push.

Metrics are served from a standalone HTTP server on a dedicated port (default 9464), separate from the main API. Serving them as a route on the API port forced an awkward trade-off: behind authentication a scraper needs an API token, while making the route public exposes it to every API consumer. A separate port avoids both — collectors scrape without API authentication, and the endpoint stays unreachable from the regular API surface and isolatable at the network layer. By default it binds to loopback, so it is not exposed off-host unless an operator opts in.

Resolves RHAIENG-5156.

How

Dependency: add opentelemetry-exporter-prometheus (pulls in prometheus-client).
setup_telemetry() (src/ogx/telemetry/__init__.py): builds a list of metric readers — the existing OTLP PeriodicExportingMetricReader when OTEL_EXPORTER_OTLP_ENDPOINT is set, plus a PrometheusMetricReader when OGX_METRICS_ENDPOINT_ENABLED is truthy. Both attach to a single global MeterProvider, so the two export paths run independently. This runs at import time and only registers the reader; it does not open a port.
start_metrics_server() (src/ogx/telemetry/__init__.py): binds the standalone scrape server via prometheus_client.start_http_server(). It is called from the server run path (create_app), not at import, so commands that merely import the telemetry module (e.g. ogx stack list-deps) do not open a network port. It serves the default Prometheus registry that PrometheusMetricReader writes to. Because it is not part of the FastAPI app, it bypasses auth by construction and never touches RequestMetricsMiddleware. A non-integer OGX_METRICS_PORT fails startup rather than silently falling back to the default.

Configuration

Env var	Default	Purpose
`OGX_METRICS_ENDPOINT_ENABLED`	unset (off)	Enable the scrape server (`1`/`true`/`yes`/`on`).
`OGX_METRICS_PORT`	`9464`	Port the scrape server listens on (non-integer fails startup).
`OGX_METRICS_HOST`	`127.0.0.1`	Bind address. Set to `0.0.0.0` (or a specific address) to expose off-host.

Acceptance criteria

When enabled, OGX exposes all existing OTel metrics in Prometheus exposition format
The endpoint does not require API authentication
Enabled/disabled via OGX_METRICS_ENDPOINT_ENABLED
Metrics are not exposed on the main API port to regular API consumers
Binds to loopback by default; off-host exposure is explicit opt-in
OTLP push continues to work independently (both paths active simultaneously)
Existing unit and integration tests continue to pass
New unit test validates Prometheus format output

Test plan

Unit tests (tests/unit/telemetry/test_prometheus_metrics.py): env-flag parsing, port parsing (default / override / invalid raises), and Prometheus-format exposition with labels and values.

uv run pytest tests/unit/telemetry/ -q
# 77 passed

Verified the bind is deferred: importing ogx.telemetry with OGX_METRICS_ENDPOINT_ENABLED=1 does not open the port (the list-deps path), while calling start_metrics_server() does and scrapes return 200.

Integration tests (tests/integration/inspect/test_metrics_endpoint.py, server mode): scrape the live scrape server on its dedicated port over raw HTTP, assert Prometheus format and ogx_requests_total, no-auth access, and that the API port returns 404 for /v1/metrics (metrics live off the API). scripts/integration-tests.sh sets OGX_METRICS_ENDPOINT_ENABLED for native server-mode runs; the tests skip otherwise.

uv run --no-sync ./scripts/integration-tests.sh \
  --stack-config server:ci-tests --setup gpt \
  --file tests/integration/inspect/test_metrics_endpoint.py
# 3 passed

🤖 Generated with Claude Code

rhdedgar

Nice, this covers all the criteria from RHAIENG-5156. +1

mergify · 2026-06-09T10:55:23Z

This pull request has merge conflicts that must be resolved before it can be merged. @cdoern please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

derekhiggins · 2026-06-17T16:57:12Z

Looking good but I think we need changes around Auth and the Serviing port number

If auth is enabled nothing will be able to scrape it unless is is configured with a token which could be problematic (at the very least would add complications)

And if we disable Auth for /metrics the then we need to expose metrics on a separate port so that it is not exposed to the same users accessing the regular API

OGX previously exported metrics only through OTLP push to an OTel Collector. This adds an optional Prometheus scrape endpoint so scrape-based monitoring systems can collect the existing metrics. When OGX_PROMETHEUS_ENABLED is set, setup_telemetry() attaches a PrometheusMetricReader to the MeterProvider alongside the existing OTLP reader, and the Inspect API serves all metrics at /v1/metrics in Prometheus exposition format. The endpoint opts out of authentication via PUBLIC_ROUTE_KEY, returns 404 when disabled, and is excluded from RequestMetricsMiddleware. The OTLP push path continues to work independently, so both export paths can run at once. Adds unit tests covering the Prometheus format and endpoint behavior, and a server-mode integration test that scrapes the live endpoint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Charlie Doern <cdoern@redhat.com>

Move the Prometheus scrape endpoint off the main API and onto a standalone HTTP server bound to its own port (OGX_PROMETHEUS_PORT, default 9464; OGX_PROMETHEUS_HOST, default 0.0.0.0). Serving metrics as a public route on the API port forced an awkward trade-off: behind authentication a scraper needs an API token, while making the route public exposes it to every API consumer. A separate port avoids both, letting collectors scrape without API authentication while keeping the endpoint unreachable from the regular API surface and isolatable at the network layer. setup_telemetry() now starts the scrape server via prometheus_client.start_http_server() when OGX_PROMETHEUS_ENABLED is truthy, alongside the existing OTLP push path. The in-API /v1/metrics route is removed from the Inspect router and no longer needs an auth bypass or exclusion from RequestMetricsMiddleware. Tests and documentation are updated accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Charlie Doern <cdoern@redhat.com>

rhdedgar

I think the new changes look good. The two integration test job issues don't appear to be related.

derekhiggins · 2026-06-26T09:24:31Z

+            # Default to all interfaces so a scraper on another host/pod can reach it;
+            # operators can pin the bind address via OGX_PROMETHEUS_HOST.
+            port = _prometheus_port()
+            host = os.environ.get("OGX_PROMETHEUS_HOST", "0.0.0.0").strip() or "0.0.0.0"  # noqa: S104


Default bind address should be 127.0.0.1 - As its a Safer upstream default.
Users and Downstream deployments can override explicitly with OGX_PROMETHEUS_HOST=0.0.0.0 if required, this will ensure metrics are only available locally unless required

Also OGX_PROMETHEUS_HOST and OGX_PROMETHEUS_PORT seems misleading, Prometheus isn't being started, maybe OGX_METRICS_HOST and OGX_METRICS_PORT ?

metrics wont be listening on IPv6 by default, maybe this is ok, just mentioning

derekhiggins · 2026-06-26T09:25:51Z

+            value=raw,
+            default=_DEFAULT_PROMETHEUS_PORT,
+        )
+        return _DEFAULT_PROMETHEUS_PORT


I don't think this should fallback to _DEFAULT_PROMETHEUS_PORT, if something has been misconfigured the server shouldn't start

derekhiggins · 2026-06-26T09:26:55Z

+
+def _is_prometheus_enabled() -> bool:
+    """Return True if the Prometheus scrape endpoint is enabled via environment."""
+    return os.environ.get("OGX_PROMETHEUS_ENABLED", "").strip().lower() in ("1", "true", "yes", "on")


Same naming problem here as with port, we're not enabling prometheus, we're enabling metrics, maybe OGX_METRICS_ENDPOINT_ENABLED ?

derekhiggins · 2026-06-26T09:28:13Z

+            metric_readers.append(PrometheusMetricReader())
+            start_http_server(port=port, addr=host)
+            logger.info(
+                "OpenTelemetry Prometheus metrics reader configured, scrape endpoint exposed",


Could we disable this code during list-deps ?

dhiggins@dhiggins-thinkpadp1gen7:~/workarea/ogx$ OGX_PROMETHEUS_ENABLED=1 uv run ogx stack list-deps --format uv config-simple.yaml INFO 2026-06-26T09:14:35.068961Z ogx.telemetry:106 OpenTelemetry Prometheus metrics reader configured, scrape endpoint exposed category=telemetry host=0.0.0.0 port=9464 uv pip install pymongo aiosqlite scikit-learn 'mcp>=1.23.0,<2.0' matplotlib pillow 'pypdf>=6.13.0' 'fonttools>=4.60.2' redis faiss-cpu asyncpg anthropic markitdown[all] chardet pandas sqlalchemy[asyncio] aiosqlite fastapi fire httpx uvicorn opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-exporter-otlp-proto-grpc

mergify · 2026-06-29T19:10:32Z

This pull request has merge conflicts that must be resolved before it can be merged. @cdoern please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

rhdedgar · 2026-06-29T19:29:45Z

Now that issue 6197 has been fixed, those two failing Integration Tests should succeed the next time they run.

Rework the metrics scrape endpoint per review: - Rename the environment variables to reflect that this is a generic metrics endpoint, not a Prometheus deployment: OGX_PROMETHEUS_ENABLED becomes OGX_METRICS_ENDPOINT_ENABLED, OGX_PROMETHEUS_PORT becomes OGX_METRICS_PORT, and OGX_PROMETHEUS_HOST becomes OGX_METRICS_HOST. - Default the bind address to loopback (127.0.0.1) instead of all interfaces, so metrics are not exposed off-host unless an operator explicitly opts in by setting OGX_METRICS_HOST. - Fail fast when OGX_METRICS_PORT is not a valid integer rather than silently falling back to the default, so a misconfiguration surfaces at startup. - Defer binding the scrape server's port to start_metrics_server(), invoked from the server run path, so commands that merely import the telemetry module (such as ogx stack list-deps) no longer open a network port. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Charlie Doern <cdoern@redhat.com>

cdoern · 2026-06-30T17:50:02Z

Thanks @derekhiggins — addressed all four in a671a109a:

Default bind address → now 127.0.0.1. Loopback by default so metrics aren't exposed off-host; operators opt in explicitly with OGX_METRICS_HOST=0.0.0.0 (or a specific address).
Env var naming → renamed to drop "Prometheus", since we're enabling a metrics endpoint, not launching Prometheus: OGX_METRICS_ENDPOINT_ENABLED, OGX_METRICS_HOST, OGX_METRICS_PORT.
Invalid port → no longer falls back to the default; a non-integer OGX_METRICS_PORT now raises and fails startup.
list-deps starting the server → the port bind moved out of import-time setup_telemetry() into start_metrics_server(), which is only called from the server run path (create_app). Importing the telemetry module (as list-deps does) now just registers the metric reader and never opens a port.

On the IPv6 note: with the loopback default it binds IPv4 127.0.0.1 only. That seemed like the right conservative default; anyone needing IPv6/dual-stack can set OGX_METRICS_HOST accordingly (e.g. ::). Happy to revisit if you'd prefer different behavior.

…heus-metrics

derekhiggins · 2026-07-01T09:59:23Z

+    except ValueError as e:
+        raise ValueError(f"Failed to parse OGX_METRICS_PORT as an integer: {raw!r}") from e

-def setup_telemetry() -> None:


You moved where the http server for metrics is started but We're still setting up telemetry during list-deps, I think we can move all of this to server run path so it doesn't run during list-deps etc...

dhiggins@dhiggins-thinkpadp1gen7:~/workarea/ogx$ OGX_METRICS_ENDPOINT_ENABLED=1 uv run ogx stack list-deps --format uv config-simple.yaml INFO 2026-07-01T09:55:55.787332Z ogx.telemetry:103 OpenTelemetry metrics scrape reader configured category=telemetry uv pip install scikit-learn sqlalchemy[asyncio] pillow 'mcp>=1.23.0,<2.0' faiss-cpu markitdown[all] 'pypdf>=6.13.0' anthropic aiosqlite matplotlib 'fonttools>=4.60.2' asyncpg redis chardet pandas pymongo aiosqlite fastapi fire httpx uvicorn opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-exporter-otlp-proto-grpc

Previously setup_telemetry() ran at module import time, so any command that imported an ogx.telemetry submodule configured telemetry as a side effect. In particular `ogx stack list-deps` set up the MeterProvider and logged that the metrics scrape reader was configured, even though it never serves requests. Move configuration onto the path where a stack is actually brought up: - Call setup_telemetry() from Stack.initialize(), the shared entry point for both server and library-client modes, and drop the import-time call. Commands that only inspect configuration, such as `ogx stack list-deps`, never build a Stack and so no longer configure telemetry or open a network port. Placing it here rather than only in create_app() preserves OTLP export for library-client users, who never go through the server app factory. - Keep start_metrics_server() in create_app(), since exposing the scrape port is a server-mode concern. Because instruments in ogx.core.server.metrics are created at import, before Stack.initialize() runs, they start as OpenTelemetry proxy instruments and rebind to the real MeterProvider once it is installed, so metric collection is unaffected. setup_telemetry() now guards against reconfiguration by checking whether a real SDK MeterProvider is already installed (get_meter_provider() returns a proxy until set), which keeps a repeated Stack.initialize() in one process from registering a duplicate Prometheus collector. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Charlie Doern <cdoern@redhat.com>

derekhiggins · 2026-07-01T16:38:43Z


-        # Create and set global MeterProvider
-        provider = MeterProvider(resource=resource, metric_readers=[reader])
+        provider = MeterProvider(resource=resource, metric_readers=metric_readers)


Hmm, looks like there could be a problem here when used with opentelemetry-instrument you see the following log on startup
WARNING 2026-07-01T16:31:27.632525Z Overriding of current MeterProvider is not allowed category=uncategorized

The opentelemetry-instrument bootstrap already handles OTLP export, so setup_telemetry()'s provider never takes effect on /metrics (we see some general process metrics but no ogx metrics)

Fixed in cdc3edd6. setup_telemetry() now detects an already-installed MeterProvider (the opentelemetry-instrument case) and adds the Prometheus reader to it via add_metric_reader() instead of installing a competing provider. Plain ogx run creates one as before.

Verified live under opentelemetry-instrument: no override warning, and /metrics exposes ogx_requests_total. Added unit tests for both branches.

…metry-instrument When ogx is launched with `opentelemetry-instrument` (the documented way to run it with telemetry), the auto-instrumentation installs the global MeterProvider and owns OTLP export. setup_telemetry() then tried to install its own provider, which OpenTelemetry refuses ("Overriding of current MeterProvider is not allowed"), so the provider carrying the Prometheus reader never took effect and the scrape endpoint exposed no ogx metrics. Detect the already-installed provider and add the Prometheus scrape reader to it via MeterProvider.add_metric_reader() instead of installing a competing one. ogx meters are already bound to that global provider, so they now reach the scrape endpoint. When no provider exists (plain `ogx run`), a new one is created as before. Also move the single entry point onto the stack bring-up path so non-serving commands stay unaffected: initialize_telemetry() (setup + scrape server start, guarded to run once) is called from Stack.initialize(), which both server and library modes reach but `ogx stack list-deps` does not. This replaces the separate calls from create_app() and keeps src/ogx/core/stack.py under the file size limit. Verified against a live server: with `opentelemetry-instrument ogx run`, the override warning is gone and /metrics exposes ogx_requests_total; plain `ogx run` is unchanged. Adds unit tests for both provider branches and initialize_telemetry idempotency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Charlie Doern <cdoern@redhat.com>

… into rhaieng-5156-prometheus-metrics

cdoern force-pushed the rhaieng-5156-prometheus-metrics branch from 838dca0 to e451304 Compare June 4, 2026 19:59

rhdedgar approved these changes Jun 5, 2026

View reviewed changes

mergify Bot added the needs-rebase label Jun 9, 2026

cdoern and others added 2 commits June 25, 2026 09:07

cdoern force-pushed the rhaieng-5156-prometheus-metrics branch from e451304 to 1c49978 Compare June 25, 2026 13:59

mergify Bot removed the needs-rebase label Jun 25, 2026

cdoern changed the title ~~feat(telemetry): expose OTel metrics via Prometheus /v1/metrics endpoint~~ feat(telemetry): expose OTel metrics via a Prometheus scrape endpoint Jun 25, 2026

cdoern requested a review from rhdedgar June 25, 2026 17:39

rhdedgar approved these changes Jun 25, 2026

View reviewed changes

derekhiggins reviewed Jun 26, 2026

View reviewed changes

rhdedgar mentioned this pull request Jun 26, 2026

ci: TypeScript streaming integration tests fail with "Premature close" on Node.js 22.23.0 #6197

Closed

2 tasks

mergify Bot added the needs-rebase label Jun 29, 2026

Merge remote-tracking branch 'upstream/main' into rhaieng-5156-promet…

193f5c6

…heus-metrics

mergify Bot removed the needs-rebase label Jun 30, 2026

derekhiggins reviewed Jul 1, 2026

View reviewed changes

cdoern marked this pull request as ready for review July 1, 2026 14:22

cdoern requested review from bbrowning, franciscojavierarceo, leseb, mattf, raghotham and skamenan7 as code owners July 1, 2026 14:22

Merge branch 'main' into rhaieng-5156-prometheus-metrics

98d2f54

derekhiggins reviewed Jul 1, 2026

View reviewed changes

cdoern and others added 2 commits July 1, 2026 13:46

Merge remote-tracking branch 'origin/rhaieng-5156-prometheus-metrics'…

b75fd4c

… into rhaieng-5156-prometheus-metrics

franciscojavierarceo approved these changes Jul 1, 2026

View reviewed changes

franciscojavierarceo added this pull request to the merge queue Jul 1, 2026

Merged via the queue into ogx-ai:main with commit 68cf84e Jul 1, 2026
76 checks passed

Uh oh!

Conversation

cdoern commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Configuration

Acceptance criteria

Test plan

Uh oh!

rhdedgar left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jun 9, 2026

Uh oh!

derekhiggins commented Jun 17, 2026

Uh oh!

rhdedgar left a comment

Choose a reason for hiding this comment

Uh oh!

derekhiggins Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

derekhiggins Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

derekhiggins Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

derekhiggins Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jun 29, 2026

Uh oh!

rhdedgar commented Jun 29, 2026

Uh oh!

cdoern commented Jun 30, 2026

Uh oh!

derekhiggins Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

derekhiggins Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

cdoern Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cdoern commented Jun 4, 2026 •

edited

Loading